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1.  Introduction 


Many  new  systems  are  being  developed  as  part  of  the  Army's  Future  Combat  Systems  (FCS).  To 
determine  the  most  effective  and  efficient  way  to  integrate  these  new  systems  within  the  future 
force,  the  U.S.  Army  Research  Laboratory’s  (ARL’s)  Human  Research  and  Engineering  Direc¬ 
torate  is  using  predictive  modeling  to  analyze  the  workload  of  FCS  operations.  This  analysis  is 
part  of  the  Robotics  Collaboration  Army  Technology  Objective  in  which  Soldier  workload  models 
of  individual  systems  are  being  developed  with  the  intent  to  be  integrated  into  one  complex  model. 
This  model  will  enable  the  investigation  of  Soldier  workload  as  well  as  how  these  Soldiers  and 
systems  can  more  effectively  combine  their  efforts  to  accomplish  a  mission.  FCS-equipped  brigade 
combat  teams  (BCTs)  consist  of  a  family  of  advanced,  networked  air-  and  ground-based  maneuver, 
maneuver  support,  and  sustainment  systems  that  will  include  manned  and  unmanned  platforms. 

The  FCS  BCT  will  rely  heavily  on  unmanned  systems  to  enable  the  “quality  of  firsts”  (see  first, 
understand  first,  act  first,  and  finish  decisively)  by  performing  such  missions  as  reconnaissance, 
surveillance,  target  acquisition,  security,  and  communications  relay.  In  fact,  at  least  1 1  types  of 
unmanned  aerial  vehicles  (UAVs)  were  committed  to  Operation  Iraqi  Freedom,  demonstrating  the 
current  need  for  the  capabilities  that  UAVs  can  provide.  The  robotics  non-commissioned  officer 
duties  within  the  FCS  BCT  include  at  least  coordination  of  and  possibly  operation  of  multiple 
unmanned  systems.  Given  the  complexity  of  the  future  operating  environment,  the  operation  of 
multiple  unmanned  systems  will  often  occur  simultaneously.  Questions  to  be  addressed  include 
(a)  how  many  unmanned  systems  can  a  Soldier  effectively  operate  simultaneously,  and  (b)  what 
level  of  autonomy  is  required  to  concurrently  operate  multiple  systems  effectively?  To  investigate 
the  possible  effects  on  performance  of  simultaneously  operating  multiple  systems,  a  workload 
model  was  developed  to  examine  operator  performance  while  as  many  as  three  micro-air  vehicles 
(MAVs)  were  operated. 

1.1  System  Description 

The  FCS  Class  I  surrogate  system  (MAV)  is  a  lift-augmented  ducted  fan  UAV  capable  of  vertical 
take-off  and  landing.  The  MAV  provides  near  real-time  electro-optical  (EO)  or  infrared  (IR)  full 
motion  video  to  support  situational  awareness  (SA)  and  understanding.  It  can  accomplish  unique 
military  missions,  particularly  with  regard  to  flight  operations  in  restricted  environments.  The 
MAV  system  includes  two  air  vehicles,  camera  sensors  (one  EO  and  one  IR)  an  operator  control 
unit  (OCU),  a  ground  data  terminal,  and  associated  ground  support  equipment.  Figures  1  through 
4  display  the  different  components  of  the  MAV.  The  OCU  and  its  associated  graphical  user 
interface  (GUI)  will  allow  the  operator  to  train,  plan,  program,  execute,  and  record  MAV  missions. 
Operators  will  direct  the  air  vehicle  (AV)  during  the  mission,  as  opposed  to  manually  flying  the 
AV  by  using  a  touch  screen.  Flexibility  in  the  OCU  and  GUI  is  important  and  must  allow  an 
operator  to  dynamically  re-task  the  AV  (assume  manual  control)  by  touching  the  screen  and 
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directing  it  to  fly  to  the  point  or  in  the  direction  touched.  The  system  will  provide  the  small  unit 
with  militarily  useful  real-time  combat  information  of  difficult-to-observe  or  distant  areas  or 
objects.  The  system  will  also  be  employable  in  a  variety  of  war-fighting  environments  (e.g.,  in 
complex  topologies  such  as  mountainous  terrain,  heavily  forested  areas,  urban  areas,  confined 
spaces,  and  high  concentrations  of  civilians).  The  initial  MAV  technology  development  program 
focused  on  the  technologies  and  components  required  to  enable  flight  at  small  scales,  including 
flight  control,  power  and  propulsion,  navigation  and  communications. 


Figure  1.  MAV. 


Figure  2.  MAV  dismounted  control  unit. 
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Figure  3.  Assembly  of  the  MAY. 


Figure  4.  MAY  in  a  backpack. 


1.2  Purpose 

The  purpose  of  this  study  is  to  assess  the  extent  to  which  MAV  operators  can  simultaneously 
operate  as  many  as  three  MAVs  in  varying  modes  of  operation  (manual  and  autonomous).  We 
accomplished  this  by  varying  the  number  of  MAVs  and  their  mode  of  operation  (manual  and 
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autonomous)  and  by  analyzing  their  associated  workloads.  These  models  of  operator  workload 
could  lead  to  improved  system  design  or  crew  configuration  changes  as  well  as  the  development  of 
tactics,  techniques,  and  procedures  that  may  save  the  Army  time,  money,  and  resources. 


2.  Method 


Modeling  and  simulation  allow  researchers  to  establish  a  relationship  between  events  and  behaviors 
while  evaluating  human  performance  without  actually  using  human  participants.  In  fact,  a  strength 
of  modeling  and  simulation  is  the  ability  to  conduct  research  on  developmental  systems  such  as 
those  proposed  for  the  FCS.  The  task  analysis  and  modeling  methodology  used  to  detennine  the 
workload  associated  with  operating  multiple  unmanned  systems  are  described  as  follows.  An 
analysis  of  MAV  operator  tasks  was  performed  which  resulted  in  a  detailed  task  list.  Modeling 
was  then  used  to  develop  a  graphical  representation  of  the  flow  of  tasks  as  well  as  the  assignment 
of  workload  values  for  each  task.  Finally,  the  MAV  model  was  executed  to  examine  the  workload 
associated  with  operating  multiple  unmanned  systems. 

2.1  Task  Analysis 

We  performed  a  task  analysis  of  the  MAV  by  observing  and  interviewing  subject  matter  experts 
(SMEs)  and  Soldiers  operating  the  system  during  a  functionality  assessment.  These  SMEs  were 
the  Soldier  Battle  Lab1  contractors  of  the  system  as  well  as  two  trained  E-6s.  All  contractors  and 
Soldiers  completed  a  40-hour  operator  training  course  of  the  system  at  Honeywell2.  The  completed 
task  list  consists  of  the  tasks  required  to  operate  the  MAV  in  manual  and  autonomous  modes. 
Finally,  participation  in  and  observation  of  MAV  functionality  assessment  occurred  at  Fort 
Benning,  Georgia.  These  tasks  are  documented  in  appendix  A. 

The  focus  of  this  study  was  to  examine  tasks  associated  with  flying  the  MAV;  therefore,  tasks  for 
launching  and  landing  the  vehicle  are  not  included.  The  operator  perfonned  three  functions  for 
each  vehicle:  manually  adjusting  the  vehicle,  monitoring  video  feedback  from  the  vehicle,  and 
monitoring  the  system  status  of  the  vehicle.  Additionally,  the  operator  performed  communication 
tasks  and  received  vibratory  alerts.  When  the  vehicle  was  operated  in  autonomous  mode,  the 
manual  adjustment  function  was  not  performed.  The  task  list  in  appendix  A  includes  the  functions, 
the  tasks  associated  with  each  function,  and  the  mean  task  time  for  each  task. 

2.2  Modeling  Approach 

ARL’s  Human  Research  and  Engineering  Directorate  has  developed  a  modeling  tool  called  the 
Improved  Performance  Research  Integration  Tool  (IMPRINT  Version  7)  to  assess  task  and 
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workload  demands  and  to  evaluate  human  and  system  performance  (IMPRINT,  2007). 

IMPRINT  incorporates  workload  theory,  which  states  that  every  task  a  human  performs  requires 
some  demand  on  attentional  resources.  IMPRINT  is  used  to  assign  values  to  the  amount  of 
effort  used  to  perform  a  task  (Mitchell,  2000). 

The  “goal-oriented”  module  of  IMPRINT  was  used  to  analyze  operator  workload  for  the  MAV. 
This  allowed  the  analyst  to  use  advanced  modeling  capabilities  such  as  tactical  branching  and 
variables  to  execute  conditions.  Additionally,  the  goal-oriented  module  uses  the  visual,  auditory, 
cognitive,  and  psychomotor  (VACP)  workload  methodology.  Goal  orientation  allows  the  mission 
to  be  analyzed  (in  this  case,  operating  the  MAV),  to  be  described  as  a  series  of  goals.  A  goal 
matrix  is  then  developed  to  describe  the  interactions  of  the  competing  goals. 

The  workload  methodology  associated  with  goal  orientation  is  called  VACP.  For  each  task  that 
the  operator  performs,  the  modeler  assigns  workload  in  each  of  the  four  resource  channels  on  a 
scale  from  0  through  7.  The  workload  scales  are  shown  in  appendix  B.  As  tasks  are  completed, 
the  demand  on  each  resource  is  summed  for  all  the  tasks  being  simultaneously  executed.  One  can 
then  view  the  times  when  the  workload  score  is  more  than  7  as  periods  of  overload  in  that  resource. 

Additionally,  the  workload  across  the  four  resources  is  sometimes  added  to  calculate  an  overall 
workload  value.  An  overall  summation  of  resource  channels  greater  than  40  has  been  bench- 
marked  as  high  workload  (Mitchell,  2003;  Mitchell  et  ah,  2003;  Wojciechowski  et  ah,  2001; 
Wojciechowski,  2006).  The  sum  of  all  resource  channels  at  a  maximum  for  a  single  task  would  be 
28  (four  channels  at  maximum  of  7).  A  score  of  40  indicates  that  more  than  one  resource  channel 
is  well  beyond  the  maximum  of  7,  meaning  that  several  channels  are  overloaded  in  more  than  one 
task.  For  the  purposes  of  this  study,  workload  was  considered  overloaded  if  any  one  channel  was 
greater  than  7  or  the  summation  of  the  four  workload  channels  was  greater  than  40. 

The  tasks  developed  in  the  previously  described  task  analysis  section  were  modeled  in  IMPRINT 
to  represent  the  tasks  of  operating  the  MAV  aircraft.  This  consisted  of  three  major  functions  for 
each  MAV  previously  described.  Also  included  were  the  two  functions  that  the  operator  performs 
in  conjunction  with  operating  the  aircraft,  independent  of  the  number  of  aircraft  being  operated: 
UAV  communication  and  receiving  vibratory  alert. 

Each  of  the  functions  was  configured  as  a  separate  goal  in  the  IMPRINT  software.  The  goals 
were  given  priorities  and  the  interaction  between  the  goals  was  controlled  by  the  goal  action 
matrix.  Figure  5  shows  the  goals  in  order  of  priority.  The  goal  action  matrix  controls  when  tasks 
happen  and  for  how  long,  based  on  priority.  In  this  case,  the  objective  was  to  determine  the 
workload  when  all  tasks  are  completed.  The  goal  action  matrix  was  set  so  that  no  tasks  were 
interrupted  or  suspended.  Therefore,  the  priority  of  the  goals  had  no  impact  on  the  operation  of 
these  tasks. 
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11 

VibAlert==1 ; 

Receive  Vibratory  Alert 

Description  of  Selected  Goal: 
Adjust  flight  of  MAV 


Cancel 


,J  Help 


Figure  5.  Goal  management  screen  indicating  goal  priority. 

The  goals  are  triggered,  based  on  three  scenarios  developed  for  operation  of  a  MAV,  which  used  a 
basic  diamond  configuration  shown  in  figure  6.  In  the  first  mission,  the  MAV  launches  from  the 
Home  Waypoint  (WP),  proceeds  to  WP  A  and  executes  a  “fly-by”.  It  then  proceeds  to  WP  B  at 
which  time,  it  circles  or  hovers  over  WP  B.  Finally,  the  AV  follows  a  road  heading  toward  WP  C. 
Before  reaching  WP  C,  the  AV  receives  a  vibratory  alert  indicating  low  fuel  at  which  point,  the 
AV  proceeds  directly  to  the  home  WP  and  lands.  Mission  2  used  the  same  diamond  configuration; 
however,  the  AV  starts  at  the  home  WP  and  proceeds  in  a  counter-clockwise  direction  toward  WP 
C,  bypassing  WP  C’s  location  and  proceeding  to  the  point  the  AV  left  the  curvy  road  in  Mission  1. 
The  AV  completes  Mission  2  by  following  the  curvy  road  to  WP  B,  then  to  WP  A  and  finally, 
back  to  the  Home  WP.  In  Mission  3,  the  AV  launched  from  the  home  WP,  climbed  in  altitude  and 
hovered  above  the  entire  area  of  operation  (AO).  About  three  quarters  of  the  way  through  the 
flight,  a  vibratory  alert  is  given,  indicating  that  additional  surveillance  is  needed  at  WP  A.  The  AV 
then  proceeds  to  WP  A,  descends  for  a  closer  look,  and  then  returns  to  the  Home  WP. 

Although  it  may  seem  that  workload  would  be  the  same  for  Scenarios  1  and  2  because  they  are 
the  same  scenario  in  reverse  order,  changing  the  order  in  which  goals  are  triggered  will  alter  the 
timing  of  specific  tasks.  This  will  give  different  workload  peaks  and  transitions  than  are  seen 
with  Scenario  1 . 
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2.3  Experimental  Design 
2.3.1  Independent  Variables 

Two  independent  variables  were  used  in  this  design.  One  was  the  number  of  manual  MAVs  that 
the  operator  was  controlling  and  the  other  was  the  number  of  autonomous  MAVs  the  operator  was 
controlling.  A  minimum  of  one  vehicle  and  a  maximum  of  three  were  chosen  for  this  investigation. 
This  design  led  to  nine  possible  combinations  of  MAVs  in  manual  and  automatic.  Each  of  these 
combinations  (shown  in  table  1)  represented  a  condition  tested  in  the  model. 


Tablet.  Nine  conditions  modeled 


Mode 

No  MAVs  in 
Manual 

One  MAV  in 
Manual 

Two  MAVs  in 
Manual 

Three  MAVs 
in  Manual 

No  MAVs  in  Autonomous 

X 

X 

X 

One  MAV  in  Autonomous 

X 

X 

X 

Two  MAVs  in  Autonomous 

X 

X 

Three  MAVs  in  Autonomous 

X 
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2.3.2  Dependent  Variables 


The  percent  time  of  the  mission  that  the  MAV  operator  is  in  mental  demand  overload  was  chosen 
as  the  dependent  measure  for  this  analysis.  The  mental  demand  is  measured  in  the  four  resource 
channels:  visual,  auditory,  cognitive,  and  psychomotor.  The  channels  are  added  for  an  overall 
workload  score  and  the  percent  time  in  overall  workload  was  measured.  For  this  experiment,  the 
analysis  focused  on  visual,  cognitive,  and  overall  demand  because  the  tasks  had  very  limited 
auditory  and  psychomotor  demand.  The  three  variables  of  interest  are  percent  time  the  operator 
had  visual  demand  greater  than  7,  the  percent  of  time  the  operator  had  a  cognitive  demand  greater 
than  7,  and  the  percent  of  time  that  the  operator  had  an  overall  workload  demand  greater  than  40. 

2.4  Procedure 

Since  three  scenarios  were  available,  all  possible  combinations  of  scenarios  for  each  condition 
were  tested  in  order  to  ensure  that  the  results  were  not  biased  by  the  scenario  chosen.  The  con¬ 
ditions  tested  were  balanced  in  terms  of  scenario  because  the  scenario  was  equally  distributed  in 
each  condition.  This  eliminated  the  effect  of  scenario  as  a  variable.  Table  2  shows  the  list  of  the 
possible  combinations.  MS1AS2AS3  is  equivalent  to  one  MAV  in  manual  flying  scenario  1,  one 
MAV  in  autonomous  flying  scenario  2,  and  one  MAV  in  autonomous  flying  scenario  3.  Fifty 
model  runs  were  completed  for  each  combination  to  ensure  that  the  variability  exhibited  was  from 
the  model  itself  and  not  from  the  random  number  seed  chosen. 

2.5  Data  Analysis 

IMPRINT  provides  output  files  that  summarize  the  workload  changes  over  time.  These  files  were 
manipulated  via  Excel '  macros  to  calculate  the  percent  time  each  model  run  was  in  overload  as 
defined  by  visual  demand  over  7,  cognitive  demand  over  7,  and  overall  demand  over  40.  These 
data  were  analyzed  with  the  Statistical  Package  for  Social  Sciences  (SPSS)4  to  determine  if  statis¬ 
tical  differences  were  present.  The  three  dependent  measures  were  analyzed  with  an  analysis  of 
variance  (ANOVA).  Post  hoc  analysis  was  completed  with  the  Least  Significant  Difference  (LSD) 
method.  ANOVA  tables  are  shown  in  appendix  C. 


Table  2.  Twenty-six  possible  combinations  of  scenario  representing  each  condition 


Mode 

No  MAVs  in  Manual 

One  MAV  in  Manual 

Two  MAVs  in  Manual 

Three  MAVs  in  Manual 

No  MAVs  in 
Autonomous 

MSI,  MS2,  MS3 

MS1MS2,  MS1MS3, 
MS2MS3 

MS1MS2MS3 

One  MAV  in 
Autonomous 

AS1,  AS2,  AS3 

MS1AS2,  MS1AS3, 
MS2AS3,  MS2AS1, 
MS3AS1,  MS3AS2 

MS1MS2AS3, 

MS1AS2MS3 

AS1MS2MS3 

Two  MAVs  in 
Autonomous 

AS1AS2,  AS1AS3, 
AS2AS3 

MS1AS2AS3, 

AS1MS2AS3, 

AS1AS2MS3 

Three  MAVs  in 
Autonomous 

AS1AS2AS3 

3Excel  is  a  registered  trademark  of  Microsoft  Corporation. 
4SPSS  is  a  registered  trademark  of  SPSS,  Inc. 
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3.  Results 


3.1  Results  by  Condition 

The  visual  workload  channel  is  in  overload  when  the  workload  level  is  greater  than  7.  Figure  7 
shows  the  mean  percent  time  in  visual  overload  for  each  condition.  The  ANOVA  indicates  sig¬ 
nificant  differences  between  conditions  F(8,441)  =  8833,/?  <  0.001.  Post  hoc  analysis  indicates 
different  conditions  as  designated  by  the  lower  case  letters  on  figure  7. 

The  cognitive  workload  channel  is  in  overload  when  the  single  channel  workload  level  is  greater 
than  7.  Figure  8  shows  the  mean  percent  time  in  cognitive  overload  for  each  condition.  The 
ANOVA  analysis  indicates  significant  differences  between  conditions  F(8,441)=7303,/?  <  0.001. 
Post  hoc  analysis  indicates  different  conditions  as  designated  by  the  lower  case  letters  on  the  chart. 

Overall  workload  is  the  sum  of  the  VACP  channels.  Overall  workload  is  considered  in  overload 
when  its  value  is  greater  than  40.  The  mean  percent  time  in  overall  workload  for  all  conditions  is 
given  in  figure  9.  The  ANOVA  shows  that  significant  differences  between  conditions  are  evident, 
F(8,441)=1741,/?  <  0.001.  Post  hoc  analysis  indicates  that  percent  time  in  overall  workload  over¬ 
load  is  approximately  zero  for  any  condition  with  one  or  two  MAVs  in  operation.  The  conditions 
when  three  MAVs  are  in  operation  are  significantly  different  from  each  other  and  all  conditions 
when  there  are  one  or  two  MAVs. 


Figure  7.  Percent  time  in  visual  overload  (>7)  by  condition. 
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Figure  8.  Percent  time  in  cognitive  overload  (>7)  by  condition. 


Manual  Manual  Manual  Auto  Auto  Auto  Manual  1  Manual  2  Manual  1 

MAV  Auto  MAVs  MAV  Auto 
Auto 


Figure  9.  Percent  time  in  overall  overload  (>40)  by  condition. 
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3.2  Results  by  Number  of  Vehicles 

ANOVAs  were  also  run  on  percent  time  in  visual  overload,  based  on  the  number  of  vehicles  in 
operation.  Significant  differences  were  shown  between  the  number  of  vehicles  in  operation 
F(2,1297)  =  2564 ,p  <  0.001.  These  data  are  presented  in  figure  10. 


Figure  10.  Percent  time  in  visual  overload  (>7)  by  number  of  vehicles. 


ANOVAs  were  also  run  on  percent  time  in  cognitive  overload  based  on  the  number  of  vehicles  in 
operation.  Significant  differences  were  shown  between  the  number  of  vehicles  in  operation 
F(2,1297)  =  70806,  p  <  0.001.  These  data  are  shown  in  figure  11. 

Significant  differences  in  overall  overload  are  shown  when  just  the  number  of  vehicles  is  analyzed, 
F(2,1297)  =  16010,/?  <  0.001.  These  data  are  shown  in  figure  12.  As  the  figure  illustrates,  there 
are  no  difference  in  the  percent  of  time  in  overall  overload  between  conditions  with  one  or  two 
MAVs. 
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3.3  Results  by  Mode  of  Control 

To  determine  the  effect  of  mode  of  operation,  conditions  with  equal  number  of  vehicles  were 
compared.  An  ANOVA  was  run  on  percent  time  in  visual  overload,  based  on  the  operation  of  a 
single  vehicle,  two  vehicles,  and  three  vehicles.  When  only  a  single  vehicle  was  operated,  there 
was  no  significant  different  in  percent  time  in  visual  overload,  based  on  operation  in  manual  or 
autonomous  mode,  as  shown  in  figure  13.  When  two  MAVs  were  operated,  there  were  significant 
differences  whether  the  vehicles  were  in  manual  or  autonomous  mode,  as  shown  in  figure  14 
(F(2,  600)  =  57. 992,  p  <  0.001).  Post  hoc  analysis  shows  that  all  three  combination  of  two 
vehicles  are  different  from  each  other.  When  three  vehicles  were  operated,  there  were  significant 
differences  for  the  mode  of  operation  as  shown  in  figure  14  (F(3,400)  =  45.669,  p  <  0.001.  Post 
hoc  analysis  indicates  that  all  combinations  of  mode  of  operation  are  different  from  each  other. 

The  same  analysis  was  perfonned,  comparing  the  percent  time  in  cognitive  overload.  ANOVA 
results  show  that  when  a  single  MAV  is  operated,  there  are  cognitive  overload  differences  between 
operation  in  manual  versus  autonomous  mode,  as  shown  in  figure  15  (F(l,300)  =  3 1 .584,  p  <  0.001. 
When  two  unmanned  vehicles  are  operated,  there  are  no  significant  differences  in  the  mode  of 
operation,  as  shown  in  figure  16.  When  three  MAVs  are  operated,  the  mode  of  operation  is  sig¬ 
nificant  as  shown  in  figure  17  (F(3,400)  =  6.222,  p  <  0.001.  Post  hoc  analysis  showed  differences 
between  all  manual  operation  and  one  or  no  vehicles  in  manual.  Two  vehicles  in  manual  are 
significantly  different  than  no  vehicles  in  manual. 

Percent  time  in  overall  overload  was  also  examined.  ANOVA  results  show  that  operation  of  a 
single  vehicle  is  not  significant  in  terms  of  the  mode  of  operation  as  shown  in  figure  18.  The 
difference  in  the  percent  time  in  overall  overload  is  significant  when  one  is  operating  two  vehicles, 
as  shown  in  figure  19  (F(2,  600)  =  4.315 ,p  =  0.014.  Post  hoc  analysis  indicates  that  in  terms  of 
percent  time  in  overall  overload,  operating  both  vehicles  in  manual  mode  is  significantly  different 
from  operating  both  vehicles  in  autonomous  mode,  but  neither  is  significantly  different  from 
operating  one  vehicle  in  manual  and  one  in  autonomous.  The  percent  time  in  overall  overload  is 
significantly  different  in  terms  of  mode  of  operation  as  shown  in  figure  20  (F(3,400)  =  43.322, 
p  <  0.001.  Post  hoc  analysis  shows  that  all  combinations  of  mode  of  operation  are  significantly 
different  from  each  other. 
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One  vehicle  in  autonomous 


mode  for  single  vehicle 

Figure  15.  Percent  time  in  overall  overload  by  mode  for  one  vehicle. 


Figure  16.  Percent  time  in  visual  overload  by  mode  for  two  vehicles. 
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Figure  17.  Percent  time  in  cognitive  overload  by  mode  for  two  vehicles. 


Figure  18.  Percent  time  in  overall  overload  by  mode  for  two  vehicles. 
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Figure  19.  Percent  time  in  visual  overload  by  mode  for  three  vehicles. 
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Figure  20.  Percent  time  in  cognitive  overload  by  mode  for  three  vehicles. 


Figure  21.  Percent  time  in  overall  overload  by  mode  for  three  vehicles. 


4.  Discussion 

The  results  from  this  study  indicate  that  operation  of  one  MAV  is  well  within  the  mental  abilities 
of  the  trained  Soldier.  This  is  validated  by  observation  of  the  MAV  operators  during  the  training 
class.  This  is  true,  regardless  of  the  mode  of  operation.  Even  though  the  percent  time  in  cognitive 
overload  is  significantly  different,  based  on  the  mode  of  operation,  the  level  is  low  (2%  to  4%)  and 
would  not  likely  cause  any  observational  or  measurable  differences  in  performance  errors  during 
these  conditions. 

Operation  of  two  MAVs  may  not  be  within  the  abilities  of  a  single  operator.  The  overall  workload 
measure  indicates  that  it  would  be  possible.  However,  examination  of  the  individual  channels  indi¬ 
cates  that  regardless  of  the  mode  of  operation,  when  a  single  Soldier  operates  two  vehicles,  his  or 
her  cognitive  workload  is  above  the  threshold  for  the  majority  of  the  mission,  80%  to  90%  of  the 
time.  Additionally,  the  visual  channel  is  overloaded  20%  to  40%  of  the  time.  It  would  appear  that 
operation  of  two  MAVs  would  quickly  lead  to  perfonnance  errors  and  possible  mission  failure.  For 
both  of  these  channels,  manual  operation  produces  a  greater  demand,  significantly  greater  for  the 
visual  channel. 
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Operation  of  three  MAVs  is  clearly  an  overload  condition,  regardless  of  the  mode  of  operation. 
Overall  workload  measures  show  that  the  operator  would  be  in  an  overload  condition  for  15%  to 
25%  of  the  mission.  Visual  workload  would  be  in  an  overload  condition  for  20%  to  40%  of  the 
mission,  and  cognitive  workload  would  again  be  in  overload  for  90%  to  91%  of  the  mission  time. 
For  all  three  measures,  manual  operation  caused  a  greater  demand;  however,  autonomous  operation 
was  still  high. 

Dixon  and  Wickens  (2003)  found  that  operation  of  two  UAVs  was  within  the  capability  of  a  single 
operator  but  that  performance  degraded.  In  their  investigation,  dual  UAV  operation  reduced 
performance  in  system  function  monitoring,  target  of  opportunity  monitoring,  and  flight  instruction 
recall.  However,  tracking  performance  and  target  report  duration  and  accuracy  were  not  impacted. 
They  also  investigated  mitigating  techniques  of  automation  and  offloading  to  auditory  resource. 
These  techniques  helped  to  mitigate  the  perfonnance  decrements.  The  mitigation  techniques  used 
by  Dixon  and  Wickens  support  the  results  of  this  study.  Automation  would  relieve  the  cognitive 
overload,  and  offloading  to  auditory  relieves  the  overload  to  the  visual  channel. 


5.  Conclusions 


The  level  of  autonomy  required  to  concurrently  operate  multiple  systems  effectively  is  an  area  that 
needs  further  research.  This  study  indicates  that  for  operation  of  more  than  one  vehicle,  a  Soldier’s 
workload  levels  are  high  enough  to  predict  that  perfonnance  errors  would  likely  occur.  Although 
enors  are  predicted,  this  investigation  does  not  predict  what  kind  of  enors  would  result.  Validation 
studies  would  help  to  determine  the  type  of  perfonnance  error  that  would  likely  occur.  Validation 
of  the  workload  numbers  and  resulting  performance  impact  would  be  necessary  to  further  this 
research. 

This  method  can  be  used  to  investigate  other  unmanned  platforms.  The  only  requirement  is 
knowledge  of  the  tasks  required  to  operate  the  system.  This  method  could  also  be  used  to  predict 
workload  issues  and  potential  performance  errors  in  systems  undergoing  development. 

Additionally,  it  is  important  to  note  that  this  investigation  only  looked  at  tasks  required  to  operate 
the  MAV.  Other  basic  Soldier  skills  required  were  not  considered  and  would  only  increase  Soldier 
workload.  Observation  during  training  validates  this  statement  because  the  Soldiers  operating  the 
MAV  were  so  occupied  that  another  Soldier  had  to  carry  their  weapons  for  them. 

Additional  studies  would  be  required  to  determine  specific  impact  on  perfonnance.  Also,  levels  of 
autonomy  would  have  to  be  examined  to  determine  what  would  be  the  optimum  level  for  operation 
of  multiple  UAVs. 
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Appendix  A.  MAY  Operator  Task  List 


Table  A-l.  MAY  operator  task  list. 


Function  Name 

Task  Name 

Mean  Time 

Manual  Adjustment  1 

Recognize  input  is  required  Ml 

00:00.1 

Manual  Adjustment  1 

Input  slow  Ml 

00:00.4 

Manual  Adjustment  1 

Input  Medium  Ml 

00:00.4 

Manual  Adjustment  1 

Input  Fast  Ml 

00:00.4 

Manual  Adjustment  1 

Input  Stop  Ml 

00:00.4 

Manual  Adjustment  1 

Wait  for  Input  Response  Ml 

00:00.1 

Monitor  Video  1 

Monitor  Video  Ml 

02:00.0 

Monitor  Video  1 

Change  Camera  Angle  Ml 

00:00.4 

Monitor  System  Status  1 

Monitor  System  Status  Ml 

00:03.0 

Manual  Adjustment  2 

Recognize  input  is  required  M2 

00:00.1 

Manual  Adjustment  2 

Input  slow  M2 

00:00.4 

Manual  Adjustment  2 

Input  Medium  M2 

00:00.4 

Manual  Adjustment  2 

Input  Fast  M2 

00:00.4 

Manual  Adjustment  2 

Input  Stop  M2 

00:00.4 

Manual  Adjustment  2 

Wait  for  Input  Response  M2 

00:00.1 

Monitor  Video  2 

Monitor  Video  M2 

02:00.0 

Monitor  Video  2 

Change  Camera  Angle  M2 

00:00.4 

Monitor  System  Status  2 

Monitor  System  Status  M2 

00:03.0 

Manual  Adjustment  3 

Recognize  input  is  required  M3 

00:00.1 

Manual  Adjustment  3 

Input  slow  M3 

00:00.4 

Manual  Adjustment  3 

Input  Medium  M3 

00:00.4 

Manual  Adjustment  3 

Input  Fast  M3 

00:00.4 

Manual  Adjustment  3 

Input  Stop  M3 

00:00.4 

Manual  Adjustment  3 

Wait  for  Input  Response  M3 

00:00.1 

Monitor  Video  3 

Monitor  Video  M3 

02:00.0 

Monitor  Video  3 

Change  Camera  Angle  M3 

00:00.4 

Monitor  System  Status  3 

Monitor  System  Status  M3 

00:03.0 

UAV  Communication 

Press  Radio  Button 

00:00.4 

UAV  Communication 

Speak 

00:05.0 

UAV  Communication 

Listen 

00:05.0 

Receive  Vibratory  Alert 

Receive  Vibratory  Alert 

00:03.0 

Receive  Vibratory  Alert 

Acknowledge  Vibratory  Alert 

00:00.1 
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Intentionally  left  blank 
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Appendix  B.  Workload  Scales 


Table  B-l.  Workload  resource  scales  and  verbal  descriptors. 


Scale 

Value 

Visual  Scale  Descriptor 

0.0 

No  Visual  Activity 

1.0 

Visually  Register/Detect  (detect  occurrence  of  image) 

3.7 

Visually  Discriminate  (detect  visual  differences) 

4.0 

Visually  Inspect/Check  (discrete  inspection/static  condition) 

5.0 

Visually  Locate/Align  (selective  orientation) 

5.4 

Visually  Track/Follow  (maintain  orientation) 

5.9 

Visually  Read  (symbol) 

7.0 

Visually  Scan/Search/Monitor  (continuous/serial  inspection,  multiple  conditions) 

Scale 

Value 

Auditory  Scale  Descriptor 

0.0 

No  Auditory  Activity 

1.0 

Detect/Register  Sound  (detect  occurrence  of  sound) 

2.0 

Orient  to  Sound  (general  orientation/attention) 

4.2 

Orient  to  Sound  (selective  orientation/attention) 

4.3 

Verify  Auditory  Feedback  (detect  occurrence  of  anticipated  sound) 

4.9 

Interpret  Semantic  Content  (speech) 

6.6 

Discriminate  Sound  Characteristics  (detect  auditory  differences) 

7.0 

Interpret  Sound  Patterns  (pulse  rates,  etc.) 

Scale 

Value 

Cognitive  Scale  Descriptor 

0.0 

No  Cognitive  Activity 

1.0 

Automatic  (simple  association) 

1.2 

Alternative  Selection 

3.7 

Sign/Signal  Recognition 

4.6 

Evaluation/Judgment  (consider  single  aspect) 

5.3 

Encoding/Decoding,  Recall 

6.8 

Evaluation/Judgment  (consider  several  aspects) 

7.0 

Estimation,  Calculation,  Conversion 

Scale 

Value 

Psychomotor  Scale  Descriptor 

0.0 

No  Psychomotor  Activity 

1.0 

Speech 

2.2 

Discrete  Actuation  (button,  toggle,  trigger) 

2.6 

Continuous  Adjustive  (flight  control,  sensor  control) 

4.6 

Manipulative 

5.8 

Discrete  Adjustive  (rotary,  vertical  thumbwheel,  lever  position) 

6.5 

Symbolic  Production  (writing) 

7.0 

Serial  Discrete  Manipulation  (keyboard  entries) 
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Intentionally  left  blank 
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Appendix  C.  ANOVA  Tables 


Table  C-l.  ANOVA  for  percent  time  in  visual  overload  by  condition. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

1622239.974 

1 

1622239.974 

276233.1 

0.000 

Condition 

415007.093 

8 

51875.887 

8833.364 

0.000 

Error 

2589.870 

441 

5.873 

Total 

2039836.937 

450 

Table  C-2.  ANOVA  for  percent  time  in  cognitive  overload  by  condition. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

1748827.760 

1 

1748827.760 

295570.5 

0.000 

Condition 

345665.637 

8 

43208.205 

7302.646 

0.000 

Error 

2609.303 

441 

5.917 

Total 

2097102.701 

450 

Table  C-3.  ANOVA  for  percent  time  in  overall  overload  by  condition. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

191167.643 

1 

191167.643 

24187.417 

0.000 

Condition 

110051.809 

8 

13756.476 

1740.533 

0.000 

Error 

3485.487 

441 

7.904 

Total 

304704.940 

450 

Table  C-4.  ANOVA  for  percent  time  in  visual  overload  by  number  of  vehicles. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

529598.263 

1 

529598.263 

11843.891 

0.000 

Condition 

229276.606 

2 

114638.303 

2563.761 

0.000 

Error 

57995.209 

1297 

44.715 

Total 

102.1834.698 

1300 

Table  C-5.  ANOVA  for  percent  time  in  cognitive  overload  by  number  of  vehicles. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

4201932.751 

1 

4201932.751 

369220.7 

0.000 

Condition 

1611628.129 

2 

805814.065 

70806.287 

0.000 

Error 

14760.566 

1297 

11.381 

Total 

1626388.695 

1300 

Table  C-6.  ANOVA  for  percent  time  in  overall  overload  by  number  of  vehicles. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

110137.693 

1 

110137.693 

15482.392 

0.000 

Condition 

227784.733 

2 

113892.366 

16010.198 

0.000 

Error 

9226.519 

1297 

7.114 

Total 

338804.611 

1300 
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Table  C-7.  ANOVA  for  percent  time  in  visual  overload  by  mode  for  one  vehicle. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

0.000 

1 

0.000 

5.167 

0.024 

Condition 

2.73E-005 

1 

2.73E-005 

0.761 

0.384 

Error 

0.011 

298 

3.59E-005 

Total 

0.011 

300 

Table  C-8.  ANOVA  for  percent  time  in  cognitive  overload  by  mode  for  one  vehicle. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

3075.681 

1 

3075.681 

1770.473 

0.000 

Condition 

54.867 

1 

54.867 

31.584 

0.000 

Error 

517.688 

298 

1.737 

Total 

3648.236 

300 

Table  C-9.  ANOVA  for  percent  time  in  overall  overload  by  mode  for  one  vehicle. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

8.74E-006 

1 

8.74E-006 

1.868 

0.173 

Condition 

8.74E-006 

1 

8.74E-006 

1.868 

0.173 

Error 

0.001 

298 

4.68E-006 

Total 

0.001 

300 

Table  C-10.  ANOVA  for  percent  time  in  visual  overload  by  mode  for  two  vehicles. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

438154.479 

1 

438154.479 

6647.779 

0.000 

Condition 

7644.560 

2 

3822.280 

57.992 

0.000 

Error 

39348.210 

597 

65.910 

Total 

533072.708 

600 

Table  C-l  1.  ANOVA  for  percent  time  in  cognitive  overload  by  mode  for  two  vehicles. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

3793399.340 

1 

3793399.340 

159958.93 

0.000 

Condition 

8.260 

2 

4.130 

0.174 

0.840 

Error 

14157.756 

597 

23.715 

Total 

4229114.346 

600 

Table  C-12.  ANOVA  for  percent  time  in  overall  overload  by  mode  for  two  vehicles. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

0.702 

1 

0.702 

222.317 

0.000 

Condition 

0.027 

2 

0.014 

4.315 

0.014 

Error 

1.884 

597 

0.003 

Total 

2.698 

600 
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Table  C-13.  ANOVA  for  percent  time  in  visual  overload  by  mode  for  three  vehicles. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

364535.226 

1 

364535.226 

17659.676 

0.000 

Condition 

2828.103 

3 

942.701 

45.669 

0.000 

Error 

8174.326 

396 

20.642 

Total 

488761.980 

400 

Table  C-14.  ANOVA  for  percent  time  in  cognitive  overload  by  mode  for  three  vehicles. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

2457416.837 

1 

2457416.837 

46329999 

0.000 

Condition 

0.990 

3 

0.330 

6.222 

0.000 

Error 

21.004 

396 

0.053 

Total 

3276596.213 

400 

Table  C-15.  ANOVA  for  percent  time  in  overall  overload  by  mode  for  three  vehicles. 


Sum  of  Squares 

df 

Mean  Square 

F 

Significance 

Intercept 

90273.999 

1 

90273.999 

14412.040 

0.000 

Condition 

814.074 

3 

271.358 

43.322 

0.000 

Error 

2480.461 

396 

6.264 

Total 

121011.901 

400 
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